Minimizing GCV/GML Scores with Multiple Smoothing Parameters via the Newton Method

The (modified) Newton method is adapted to optimize generalized cross validation (GCV) and generalized maximum likelihood (GML) scores with multiple smoothing parameters. The main concerns in solving the optimization problem are the speed and the reliability of the algorithm, as well as the invariance of the algorithm under transformations under which the problem itself is invariant. The proposed algorithm is believed to be highly efficient for the problem, though it is still rather expensive for large data sets, since its operational counts are (2/3)kn + O(n2), with k the number of smoothing parameters and n the number of observations. Sensible procedures for computing good starting values are also proposed, which should help in keeping the execution load to the minimum possible. The algorithm is implemented in Rkpack [RKPACK and its applications: Fitting smoothing spline models, Tech. Report 857, Department of Statistics, University of Wisconsin, Madison, WI, 1989] and illustrated by examples of fitting additive and interaction spline models. It is noted that the algorithm can also be applied to the maximum likelihood (ML) and the restricted maximum likelihood (REML) estimation of the variance component models. Key words, additive/interaction spline models, gradient, Hessian, invariance, Newton method, smoothing parameters, starting values AMS(MOS) subject classifications. 65D07, 65D10, 65D15, 65K10, 65V05